Leverage local links on git checkouts #4919
Merged
+69
−29
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit updates the handling of git checkouts from the database to use hardlinks if possible, speeding up this operation for large repositories significantly. As a refresher, Cargo caches git repositories in a few locations to speed up local usage of git repositories. Cargo has a "database" folder which is a bare checkout of any git repository Cargo has cached historically. This database folder contains effectively a bunch of databases for remote repos that are updated periodically. When actually building a crate Cargo will clone this database into a different location, the checkouts folder. Each rev we build (ever) is cached in the checkouts folder. This means that once a checkout directory is created it's frozen for all of time. This latter step is what this commit is optimizing. When checking out the database onto the local filesystem at a particular revision. Previously we were instructing libgit2 to fall back to a "git aware" transport which was exceedingly slow on some systems for filesystem-to-filesystem transfers. This optimization (we just forgot to turn it on in libgit2) is a longstanding one and should speed this up significantly! Closes rust-lang#4604
|
r? @matklad (rust_highfive has picked a reviewer for you, use r? to override) |
|
LGTM! @bors r+ |
|
📌 Commit 5cca4e8 has been approved by |
bors
added a commit
that referenced
this pull request
Jan 8, 2018
Leverage local links on git checkouts This commit updates the handling of git checkouts from the database to use hardlinks if possible, speeding up this operation for large repositories significantly. As a refresher, Cargo caches git repositories in a few locations to speed up local usage of git repositories. Cargo has a "database" folder which is a bare checkout of any git repository Cargo has cached historically. This database folder contains effectively a bunch of databases for remote repos that are updated periodically. When actually building a crate Cargo will clone this database into a different location, the checkouts folder. Each rev we build (ever) is cached in the checkouts folder. This means that once a checkout directory is created it's frozen for all of time. This latter step is what this commit is optimizing. When checking out the database onto the local filesystem at a particular revision. Previously we were instructing libgit2 to fall back to a "git aware" transport which was exceedingly slow on some systems for filesystem-to-filesystem transfers. This optimization (we just forgot to turn it on in libgit2) is a longstanding one and should speed this up significantly! Closes #4604
|
☀️ Test successful - status-appveyor, status-travis |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit updates the handling of git checkouts from the database to use
hardlinks if possible, speeding up this operation for large repositories
significantly.
As a refresher, Cargo caches git repositories in a few locations to speed up
local usage of git repositories. Cargo has a "database" folder which is a bare
checkout of any git repository Cargo has cached historically. This database
folder contains effectively a bunch of databases for remote repos that are
updated periodically.
When actually building a crate Cargo will clone this database into a different
location, the checkouts folder. Each rev we build (ever) is cached in the
checkouts folder. This means that once a checkout directory is created it's
frozen for all of time.
This latter step is what this commit is optimizing. When checking out the
database onto the local filesystem at a particular revision. Previously we were
instructing libgit2 to fall back to a "git aware" transport which was
exceedingly slow on some systems for filesystem-to-filesystem transfers. This
optimization (we just forgot to turn it on in libgit2) is a longstanding one and
should speed this up significantly!
Closes #4604